A simple sketching algorithm for entropy estimation

We consider the problem of approximating the empirical Shannon entropy of a high-frequency data stream when space limitations make exact computation infeasible. It is known that αdependent quantities such as the Rényi and Tsallis entropies can be estimated efficiently and unbiasedly from low-dimensional α-stable data sketches. An approximation to the Shannon entropy can be obtained from either of these quantities by taking α sufficiently close to 1. However, practical guidelines for the choice of α are lacking. We avoid this problem by going directly to the limit. We show that the projection variables used in estimating the Rényi entropy can be transformed to have a proper distributional limit as α approaches 1. The Shannon entropy can then be estimated directly from a data sketch based on this limiting distribution. We derive properties of the distribution, showing that it has a surprisingly simple characteristic function (iθ) and that the kth moment of the exponential of such a variable is k for all non-negative real values of k. These properties enable the Shannon entropy to be estimated directly from the associated data sketch as the logarithm of a simple average. We obtain the Fisher information for the statistical problem of recovering the entropy from the data sketch and hence a lower bound on the standard error of the estimated entropy. We show that our proposed estimator has theoretical statistical efficiency of 96.8% and confirm this with an empirical study. Finally we demonstrate that in order for the estimator to have 1+ ǫ coverage with high probability the sketch must have size O(1/ǫ), in agreement with theoretical bounds.